Eigenvalue-Corrected Natural Gradient Based on a New Approximation

Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC) (George et al., 2018), proposes an interpretation of viewing natural gradient update as a diagonal and corrects the inaccurate re-scaling factor in Kronecker-factored eigenbasis. Gao al. (2020) considers new approximation to gradient, which approximates Fisher information matrix (FIM) constant multiplied by product two matrices keeps trace equal before after approximation. In this work, we combine ideas these propose Trace-restricted (TEKFAC). The method not only inexact under eigenbasis, but also effective damping technique (2020). We discuss differences relationships among approximations. Empirically, our outperforms SGD with momentum, Adam, EKFAC TKFAC on several DNNs.